Skip to content

Fix spark application spans status on sql analysis failure#10981

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits intomasterfrom
adrien.boitreaud/fix-spark-app-status-on-sql-analysis-failure
Mar 31, 2026
Merged

Fix spark application spans status on sql analysis failure#10981
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits intomasterfrom
adrien.boitreaud/fix-spark-app-status-on-sql-analysis-failure

Conversation

@aboitreaud
Copy link
Copy Markdown
Contributor

@aboitreaud aboitreaud commented Mar 27, 2026 •
edited
Loading

What Does This Do

Context: SparkSession.sql() calls that throw during Catalyst analysis (e.g. AnalysisException for missing tables) fire before any Spark job/stage events: our current instrumentation never sees them, so the sparl.application span stays green.

  • Adds SparkSqlFailureAdvice on SparkSession.sql() to catch these throwables and call onSqlFailure() on the listener
  • finishApplication() now marks the app span ERROR with error.type = "Spark SQL Failed" when no job ran but a SQL failure was recorded
  • Job failures take precedence: a successful job resets lastSqlFailed, and lastJobFailed wins in finishApplication() if both are set

Motivation

Make sure that Spark application span are marked as error when the call spark.sql().show() fails on SQL analysis failure

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@aboitreaud
Track Spark SQL analysis failures to mark application spans as ERROR
824aceb 
Add lastSqlFailed tracking to AbstractDatadogSparkListener when SQL calls
(e.g. SparkSession.sql()) throw exceptions during Catalyst analysis, before
any Spark job is submitted. This ensures finishApplication() can mark the
application span as ERROR even when no job/stage/task events fire.

The error priority in finishApplication() is:
  throwable (from caller) > exitCode != 0 > lastJobFailed > lastSqlFailed

Add unit tests to verify SQL failures mark application spans as ERROR,
and that job failures take precedence over SQL failures.

Fixes: Spark application traces marked SUCCESS when SQL analysis fails
@aboitreaud
Instrument SparkSession.sql() to capture analysis failures
0910238 
Add SparkSqlFailureAdvice that intercepts SparkSession.sql() method calls
and propagates any exceptions (e.g. AnalysisException) to the listener via
the new onSqlFailure() callback.

This ensures SQL analysis failures that occur before any Spark job is
submitted are captured and can be reported as ERROR in the application span.
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Mar 27, 2026 •
edited
Loading

Benchmarks

⚠️ Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master adrien.boitreaud/fix-spark-app-status-on-sql-analysis-failure
git_commit_date 1774925260 1774944292
git_commit_sha b7fb6fc f497183
release_version 1.61.0-SNAPSHOT~b7fb6fc123 1.61.0-SNAPSHOT~f497183167
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1774946127 1774946127
ci_job_id 1553261483 1553261483
ci_pipeline_id 105199834 105199834
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-ak8o2w6b 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-ak8o2w6b 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1062105
Total [baseline] (8.845 s) : 0, 8844898
Agent [candidate] (1.062 s) : 0, 1061910
Total [candidate] (8.861 s) : 0, 8860530
section iast
Agent [baseline] (1.231 s) : 0, 1230911
Total [baseline] (9.564 s) : 0, 9564099
Agent [candidate] (1.234 s) : 0, 1234331
Total [candidate] (9.55 s) : 0, 9550444
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent iast 1.231 s 168.806 ms (15.9%)
Total tracing 8.845 s -
Total iast 9.564 s 719.201 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent iast 1.234 s 172.421 ms (16.2%)
Total tracing 8.861 s -
Total iast 9.55 s 689.914 ms (7.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.219 ms) : 0, 1219
crashtracking [candidate] (1.199 ms) : 0, 1199
BytebuddyAgent [baseline] (632.211 ms) : 0, 632211
BytebuddyAgent [candidate] (632.168 ms) : 0, 632168
AgentMeter [baseline] (29.447 ms) : 0, 29447
AgentMeter [candidate] (29.739 ms) : 0, 29739
GlobalTracer [baseline] (258.267 ms) : 0, 258267
GlobalTracer [candidate] (257.982 ms) : 0, 257982
AppSec [baseline] (32.002 ms) : 0, 32002
AppSec [candidate] (31.799 ms) : 0, 31799
Debugger [baseline] (59.9 ms) : 0, 59900
Debugger [candidate] (59.888 ms) : 0, 59888
Remote Config [baseline] (590.51 µs) : 0, 591
Remote Config [candidate] (610.303 µs) : 0, 610
Telemetry [baseline] (8.121 ms) : 0, 8121
Telemetry [candidate] (8.051 ms) : 0, 8051
Flare Poller [baseline] (4.256 ms) : 0, 4256
Flare Poller [candidate] (4.317 ms) : 0, 4317
section iast
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.204 ms) : 0, 1204
BytebuddyAgent [baseline] (799.556 ms) : 0, 799556
BytebuddyAgent [candidate] (802.217 ms) : 0, 802217
AgentMeter [baseline] (11.416 ms) : 0, 11416
AgentMeter [candidate] (11.439 ms) : 0, 11439
GlobalTracer [baseline] (247.67 ms) : 0, 247670
GlobalTracer [candidate] (248.82 ms) : 0, 248820
AppSec [baseline] (26.553 ms) : 0, 26553
AppSec [candidate] (27.318 ms) : 0, 27318
Debugger [baseline] (67.677 ms) : 0, 67677
Debugger [candidate] (67.929 ms) : 0, 67929
Remote Config [baseline] (542.924 µs) : 0, 543
Remote Config [candidate] (522.909 µs) : 0, 523
Telemetry [baseline] (10.768 ms) : 0, 10768
Telemetry [candidate] (9.668 ms) : 0, 9668
Flare Poller [baseline] (3.855 ms) : 0, 3855
Flare Poller [candidate] (3.562 ms) : 0, 3562
IAST [baseline] (25.445 ms) : 0, 25445
IAST [candidate] (25.368 ms) : 0, 25368
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1061963
Total [baseline] (11.061 s) : 0, 11061274
Agent [candidate] (1.06 s) : 0, 1059782
Total [candidate] (11.085 s) : 0, 11084641
section appsec
Agent [baseline] (1.251 s) : 0, 1250673
Total [baseline] (11.132 s) : 0, 11131991
Agent [candidate] (1.252 s) : 0, 1252443
Total [candidate] (11.136 s) : 0, 11136183
section iast
Agent [baseline] (1.244 s) : 0, 1244330
Total [baseline] (11.357 s) : 0, 11356745
Agent [candidate] (1.234 s) : 0, 1233896
Total [candidate] (11.304 s) : 0, 11304057
section profiling
Agent [baseline] (1.187 s) : 0, 1186629
Total [baseline] (11.049 s) : 0, 11048588
Agent [candidate] (1.185 s) : 0, 1185430
Total [candidate] (11.017 s) : 0, 11017459
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent appsec 1.251 s 188.71 ms (17.8%)
Agent iast 1.244 s 182.367 ms (17.2%)
Agent profiling 1.187 s 124.666 ms (11.7%)
Total tracing 11.061 s -
Total appsec 11.132 s 70.717 ms (0.6%)
Total iast 11.357 s 295.471 ms (2.7%)
Total profiling 11.049 s -12.687 ms (-0.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.06 s -
Agent appsec 1.252 s 192.66 ms (18.2%)
Agent iast 1.234 s 174.113 ms (16.4%)
Agent profiling 1.185 s 125.647 ms (11.9%)
Total tracing 11.085 s -
Total appsec 11.136 s 51.542 ms (0.5%)
Total iast 11.304 s 219.416 ms (2.0%)
Total profiling 11.017 s -67.182 ms (-0.6%) 
gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.191 ms) : 0, 1191
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (631.989 ms) : 0, 631989
BytebuddyAgent [candidate] (630.939 ms) : 0, 630939
AgentMeter [baseline] (29.451 ms) : 0, 29451
AgentMeter [candidate] (29.607 ms) : 0, 29607
GlobalTracer [baseline] (257.879 ms) : 0, 257879
GlobalTracer [candidate] (257.504 ms) : 0, 257504
AppSec [baseline] (31.982 ms) : 0, 31982
AppSec [candidate] (31.798 ms) : 0, 31798
Debugger [baseline] (60.454 ms) : 0, 60454
Debugger [candidate] (60.405 ms) : 0, 60405
Remote Config [baseline] (622.016 µs) : 0, 622
Remote Config [candidate] (619.823 µs) : 0, 620
Telemetry [baseline] (8.826 ms) : 0, 8826
Telemetry [candidate] (8.103 ms) : 0, 8103
Flare Poller [baseline] (3.501 ms) : 0, 3501
Flare Poller [candidate] (3.51 ms) : 0, 3510
section appsec
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (661.436 ms) : 0, 661436
BytebuddyAgent [candidate] (661.655 ms) : 0, 661655
AgentMeter [baseline] (12.08 ms) : 0, 12080
AgentMeter [candidate] (12.24 ms) : 0, 12240
GlobalTracer [baseline] (258.588 ms) : 0, 258588
GlobalTracer [candidate] (259.156 ms) : 0, 259156
AppSec [baseline] (177.953 ms) : 0, 177953
AppSec [candidate] (178.45 ms) : 0, 178450
Debugger [baseline] (66.487 ms) : 0, 66487
Debugger [candidate] (66.494 ms) : 0, 66494
Remote Config [baseline] (633.756 µs) : 0, 634
Remote Config [candidate] (656.355 µs) : 0, 656
Telemetry [baseline] (8.352 ms) : 0, 8352
Telemetry [candidate] (8.412 ms) : 0, 8412
Flare Poller [baseline] (3.554 ms) : 0, 3554
Flare Poller [candidate] (3.554 ms) : 0, 3554
IAST [baseline] (24.163 ms) : 0, 24163
IAST [candidate] (24.27 ms) : 0, 24270
section iast
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (808.196 ms) : 0, 808196
BytebuddyAgent [candidate] (800.268 ms) : 0, 800268
AgentMeter [baseline] (11.729 ms) : 0, 11729
AgentMeter [candidate] (11.459 ms) : 0, 11459
GlobalTracer [baseline] (249.997 ms) : 0, 249997
GlobalTracer [candidate] (248.756 ms) : 0, 248756
AppSec [baseline] (26.973 ms) : 0, 26973
AppSec [candidate] (26.728 ms) : 0, 26728
Debugger [baseline] (69.662 ms) : 0, 69662
Debugger [candidate] (69.995 ms) : 0, 69995
Remote Config [baseline] (525.433 µs) : 0, 525
Remote Config [candidate] (527.244 µs) : 0, 527
Telemetry [baseline] (10.193 ms) : 0, 10193
Telemetry [candidate] (9.724 ms) : 0, 9724
Flare Poller [baseline] (3.782 ms) : 0, 3782
Flare Poller [candidate] (3.57 ms) : 0, 3570
IAST [baseline] (25.714 ms) : 0, 25714
IAST [candidate] (25.437 ms) : 0, 25437
section profiling
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.169 ms) : 0, 1169
BytebuddyAgent [baseline] (684.674 ms) : 0, 684674
BytebuddyAgent [candidate] (684.621 ms) : 0, 684621
AgentMeter [baseline] (8.972 ms) : 0, 8972
AgentMeter [candidate] (8.97 ms) : 0, 8970
GlobalTracer [baseline] (215.452 ms) : 0, 215452
GlobalTracer [candidate] (215.257 ms) : 0, 215257
AppSec [baseline] (32.497 ms) : 0, 32497
AppSec [candidate] (32.337 ms) : 0, 32337
Debugger [baseline] (66.032 ms) : 0, 66032
Debugger [candidate] (65.283 ms) : 0, 65283
Remote Config [baseline] (575.223 µs) : 0, 575
Remote Config [candidate] (568.447 µs) : 0, 568
Telemetry [baseline] (7.788 ms) : 0, 7788
Telemetry [candidate] (7.748 ms) : 0, 7748
Flare Poller [baseline] (3.539 ms) : 0, 3539
Flare Poller [candidate] (4.282 ms) : 0, 4282
ProfilingAgent [baseline] (94.746 ms) : 0, 94746
ProfilingAgent [candidate] (94.212 ms) : 0, 94212
Profiling [baseline] (95.31 ms) : 0, 95310
Profiling [candidate] (94.76 ms) : 0, 94760
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master adrien.boitreaud/fix-spark-app-status-on-sql-analysis-failure
git_commit_date 1774925260 1774944292
git_commit_sha b7fb6fc f497183
release_version 1.61.0-SNAPSHOT~b7fb6fc123 1.61.0-SNAPSHOT~f497183167
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1774946601 1774946601
ci_job_id 1553261486 1553261486
ci_pipeline_id 105199834 105199834
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-4-vyg44qsn 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-4-vyg44qsn 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 18 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:petclinic:profiling:high_load worse
[+0.746ms; +1.776ms] or [+4.022%; +9.580%]		 worse
[+1.005ms; +2.757ms] or [+3.393%; +9.306%]		 unstable
[-44.301op/s; +10.926op/s] or [-17.745%; +4.377%]		 19.804ms 31.502ms 232.969op/s 18.543ms 29.621ms 249.656op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.181 ms) : 17997, 18365
.   : milestone, 18181,
appsec (18.78 ms) : 18594, 18965
.   : milestone, 18780,
code_origins (17.996 ms) : 17819, 18174
.   : milestone, 17996,
iast (18.14 ms) : 17957, 18322
.   : milestone, 18140,
profiling (18.692 ms) : 18507, 18878
.   : milestone, 18692,
tracing (17.759 ms) : 17584, 17933
.   : milestone, 17759,
section candidate
no_agent (18.226 ms) : 18038, 18414
.   : milestone, 18226,
appsec (18.657 ms) : 18470, 18843
.   : milestone, 18657,
code_origins (17.8 ms) : 17623, 17978
.   : milestone, 17800,
iast (17.735 ms) : 17560, 17910
.   : milestone, 17735,
profiling (20.037 ms) : 19831, 20242
.   : milestone, 20037,
tracing (17.794 ms) : 17618, 17969
.   : milestone, 17794,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.181 ms [17.997 ms, 18.365 ms] -
appsec 18.78 ms [18.594 ms, 18.965 ms] 598.362 µs (3.3%)
code_origins 17.996 ms [17.819 ms, 18.174 ms] -184.933 µs (-1.0%)
iast 18.14 ms [17.957 ms, 18.322 ms] -41.6 µs (-0.2%)
profiling 18.692 ms [18.507 ms, 18.878 ms] 511.09 µs (2.8%)
tracing 17.759 ms [17.584 ms, 17.933 ms] -422.379 µs (-2.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.226 ms [18.038 ms, 18.414 ms] -
appsec 18.657 ms [18.47 ms, 18.843 ms] 430.753 µs (2.4%)
code_origins 17.8 ms [17.623 ms, 17.978 ms] -425.828 µs (-2.3%)
iast 17.735 ms [17.56 ms, 17.91 ms] -490.874 µs (-2.7%)
profiling 20.037 ms [19.831 ms, 20.242 ms] 1.811 ms (9.9%)
tracing 17.794 ms [17.618 ms, 17.969 ms] -432.506 µs (-2.4%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.243 ms) : 1230, 1255
.   : milestone, 1243,
iast (3.175 ms) : 3132, 3219
.   : milestone, 3175,
iast_FULL (6.107 ms) : 6044, 6170
.   : milestone, 6107,
iast_GLOBAL (3.765 ms) : 3702, 3828
.   : milestone, 3765,
profiling (2.317 ms) : 2294, 2339
.   : milestone, 2317,
tracing (2.014 ms) : 1995, 2033
.   : milestone, 2014,
section candidate
no_agent (1.236 ms) : 1225, 1248
.   : milestone, 1236,
iast (3.174 ms) : 3133, 3215
.   : milestone, 3174,
iast_FULL (6.032 ms) : 5971, 6093
.   : milestone, 6032,
iast_GLOBAL (3.699 ms) : 3637, 3761
.   : milestone, 3699,
profiling (2.298 ms) : 2276, 2320
.   : milestone, 2298,
tracing (1.896 ms) : 1879, 1912
.   : milestone, 1896,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.243 ms [1.23 ms, 1.255 ms] -
iast 3.175 ms [3.132 ms, 3.219 ms] 1.933 ms (155.6%)
iast_FULL 6.107 ms [6.044 ms, 6.17 ms] 4.864 ms (391.4%)
iast_GLOBAL 3.765 ms [3.702 ms, 3.828 ms] 2.522 ms (203.0%)
profiling 2.317 ms [2.294 ms, 2.339 ms] 1.074 ms (86.4%)
tracing 2.014 ms [1.995 ms, 2.033 ms] 771.631 µs (62.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.236 ms [1.225 ms, 1.248 ms] -
iast 3.174 ms [3.133 ms, 3.215 ms] 1.938 ms (156.8%)
iast_FULL 6.032 ms [5.971 ms, 6.093 ms] 4.796 ms (387.9%)
iast_GLOBAL 3.699 ms [3.637 ms, 3.761 ms] 2.463 ms (199.2%)
profiling 2.298 ms [2.276 ms, 2.32 ms] 1.062 ms (85.9%)
tracing 1.896 ms [1.879 ms, 1.912 ms] 659.353 µs (53.3%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master adrien.boitreaud/fix-spark-app-status-on-sql-analysis-failure
git_commit_date 1774925260 1774944292
git_commit_sha b7fb6fc f497183
release_version 1.61.0-SNAPSHOT~b7fb6fc123 1.61.0-SNAPSHOT~f497183167
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1774946492 1774946492
ci_job_id 1553261489 1553261489
ci_pipeline_id 105199834 105199834
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-cwln0i3r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-cwln0i3r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.063 s) : 15063000, 15063000
.   : milestone, 15063000,
appsec (15.044 s) : 15044000, 15044000
.   : milestone, 15044000,
iast (17.828 s) : 17828000, 17828000
.   : milestone, 17828000,
iast_GLOBAL (17.857 s) : 17857000, 17857000
.   : milestone, 17857000,
profiling (15.515 s) : 15515000, 15515000
.   : milestone, 15515000,
tracing (14.783 s) : 14783000, 14783000
.   : milestone, 14783000,
section candidate
no_agent (15.235 s) : 15235000, 15235000
.   : milestone, 15235000,
appsec (15.23 s) : 15230000, 15230000
.   : milestone, 15230000,
iast (18.59 s) : 18590000, 18590000
.   : milestone, 18590000,
iast_GLOBAL (17.838 s) : 17838000, 17838000
.   : milestone, 17838000,
profiling (15.011 s) : 15011000, 15011000
.   : milestone, 15011000,
tracing (14.883 s) : 14883000, 14883000
.   : milestone, 14883000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.063 s [15.063 s, 15.063 s] -
appsec 15.044 s [15.044 s, 15.044 s] -19.0 ms (-0.1%)
iast 17.828 s [17.828 s, 17.828 s] 2.765 s (18.4%)
iast_GLOBAL 17.857 s [17.857 s, 17.857 s] 2.794 s (18.5%)
profiling 15.515 s [15.515 s, 15.515 s] 452.0 ms (3.0%)
tracing 14.783 s [14.783 s, 14.783 s] -280.0 ms (-1.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.235 s [15.235 s, 15.235 s] -
appsec 15.23 s [15.23 s, 15.23 s] -5.0 ms (-0.0%)
iast 18.59 s [18.59 s, 18.59 s] 3.355 s (22.0%)
iast_GLOBAL 17.838 s [17.838 s, 17.838 s] 2.603 s (17.1%)
profiling 15.011 s [15.011 s, 15.011 s] -224.0 ms (-1.5%)
tracing 14.883 s [14.883 s, 14.883 s] -352.0 ms (-2.3%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~f497183167, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.482 ms) : 1470, 1493
.   : milestone, 1482,
appsec (3.758 ms) : 3542, 3974
.   : milestone, 3758,
iast (2.261 ms) : 2193, 2330
.   : milestone, 2261,
iast_GLOBAL (2.313 ms) : 2244, 2382
.   : milestone, 2313,
profiling (2.1 ms) : 2044, 2155
.   : milestone, 2100,
tracing (2.082 ms) : 2028, 2135
.   : milestone, 2082,
section candidate
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (3.744 ms) : 3526, 3961
.   : milestone, 3744,
iast (2.266 ms) : 2197, 2335
.   : milestone, 2266,
iast_GLOBAL (2.308 ms) : 2239, 2377
.   : milestone, 2308,
profiling (2.112 ms) : 2056, 2168
.   : milestone, 2112,
tracing (2.076 ms) : 2023, 2129
.   : milestone, 2076,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.482 ms [1.47 ms, 1.493 ms] -
appsec 3.758 ms [3.542 ms, 3.974 ms] 2.276 ms (153.7%)
iast 2.261 ms [2.193 ms, 2.33 ms] 779.554 µs (52.6%)
iast_GLOBAL 2.313 ms [2.244 ms, 2.382 ms] 831.59 µs (56.1%)
profiling 2.1 ms [2.044 ms, 2.155 ms] 617.974 µs (41.7%)
tracing 2.082 ms [2.028 ms, 2.135 ms] 600.002 µs (40.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.479 ms [1.468 ms, 1.491 ms] -
appsec 3.744 ms [3.526 ms, 3.961 ms] 2.264 ms (153.1%)
iast 2.266 ms [2.197 ms, 2.335 ms] 786.473 µs (53.2%)
iast_GLOBAL 2.308 ms [2.239 ms, 2.377 ms] 828.537 µs (56.0%)
profiling 2.112 ms [2.056 ms, 2.168 ms] 632.707 µs (42.8%)
tracing 2.076 ms [2.023 ms, 2.129 ms] 596.821 µs (40.3%)

@aboitreaud aboitreaud marked this pull request as ready for review March 27, 2026 16:16
@aboitreaud aboitreaud requested a review from a team as a code owner March 27, 2026 16:16
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 27, 2026

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@aboitreaud aboitreaud added type: enhancement Enhancements and improvements inst: apache spark Apache Spark instrumentation labels Mar 27, 2026
pawel-big-lebowski
pawel-big-lebowski reviewed Mar 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@pawel-big-lebowski pawel-big-lebowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good to me. Would it be possible to include End to end test which:

  • creates sparkSession = SparkSession.builder()..
  • runs sparkSession.sql("""...) with table not found for example
  • and checks if the trace contains expected error attributes.

Similar tests exist in AbstractSpark32SqlTest.

@aboitreaud
Copy link
Copy Markdown
Contributor Author

aboitreaud commented Mar 31, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 bot commented Mar 31, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-03-31 09:01:46 UTC ℹ️ Start processing command /merge

2026-03-31 09:01:59 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-03-31 10:07:57 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot merged commit d40737c into master Mar 31, 2026
565 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot deleted the adrien.boitreaud/fix-spark-app-status-on-sql-analysis-failure branch March 31, 2026 10:07
@github-actions github-actions bot added this to the 1.61.0 milestone Mar 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pawel-big-lebowski pawel-big-lebowski pawel-big-lebowski approved these changes

Assignees

No one assigned

Labels

inst: apache spark Apache Spark instrumentation type: enhancement Enhancements and improvements

Projects

None yet

Milestone

1.61.0

Development

Successfully merging this pull request may close these issues.

2 participants

@aboitreaud @pawel-big-lebowski